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Analyzing multilevel data: An empirical comparison of parameter estimates of 
hierarchical linear modeling and ordinary least squares regression 
A central phenomenon of interest in higher education research is the impact of college on 
students. Scholars seek to understand how personal characteristics of students and aspects of 
their educational experiences influence students’ academic learning and growth. This learning 
chiefly takes place in the organizational settings of institutions and features of these settings can 
have substantial influences on students’ growth and development in college (Pascarella & 
Terenzini 1991, 2005; Toutkoushian & Smart, 2001). While it is widely recognized that 
institutional characteristics impact students’ growth and development, a major problem in 
assessing that impact on students is the methodological dilemmas due to the multilevel character 
of the majority of data used in such studies. 

Multilevel or hierarchical data are a common fixture in higher education. The classic 
example of multilevel data in higher education is students grouped or ‘nested’ within institutions. 
Multi-institutional datasets often contain variables that describe students as well as variables that 
describe institutions. For instance, data collected on college students may contain variables that 
describe students, such as interactions with faculty members and other students, experiences in 
coursework and extracurricular activities, as well as variables that describe institutions, such as 
sector, selectivity, and graduation rates. Additionally, even single-institution studies could have a 
hierarchical nature given the organization of postsecondary institutions. Students are nested in 
classes, majors, departments, and colleges or schools within an institution. Furthennore, it is 
common to find analyses with students nested within academic majors nested within institutions, 
where the individual, major, and institution are all the objects of interest and of observation. 
Despite the prevalence of hierarchical structures present in post-secondary educational research, 
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past studies have often failed to address them adequately in the data analysis (Burstein, 1980; 
Ethington, 1997; Pascarella & Terenzini, 1991, 2005). 

In his seminal critique of educational research, Burstein (1980) argued that existing 
statistical techniques were simply inadequate for estimating the effects of school on students. 
While Burstein’ s discussion focused on the research on school effects at the elementary and 
secondary level, the arguments and the methodological concerns he presents are also applicable 
to research focusing on the influence of college on students (Ethington, 1997). Burstein notes 
that the models used in school effects research had been single-level and based on the traditional 
linear model ordinary least squares (OLS) regression, which, he argues, does not adequately 
match the realities under investigation. Although researchers had acknowledged the hierarchical 
nature of the organization of schooling by gathering data on students, classes, and schools, the 
statistical model reflected only a single level. Burstein argued that this neglect of the hierarchical 
nature of the data gathered reflects the limitations of the existing statistical techniques at that 
time for the estimation of the linear models with nested structures rather than a conviction on the 
part of the researcher that the single-level statistical model was appropriate. There simply were 
no viable alternatives. 

Historically, two common procedures have been used when analyzing hierarchical data. 
The first procedure is to disaggregate all higher order variables to the individual level, and the 
analysis is done at the individual level. The second procedure is to aggregate the individual level 
variables to the higher level and do the analysis at the higher level. However, a number of 
conceptual and technical difficulties such as aggregation bias, misestimated standard errors, and 
heterogeneity of regression have plagued these studies (Bumstein, 1980; Raudenbush & Bryk, 
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2002). These two procedures are known as the unit of analysis problem and have plagued 
researchers in their attempt to analyze hierarchical data. 

Pascarella and Terenzini (1991) discuss the unit of analysis problem associated with the 
hierarchical nature of data in higher education and suggest that differences in the units of 
analysis used in studies examining similar phenomena may have contributed to the lack of 
consistency in findings in the influence of college on students. Pascarella and Terenzini reviewed 
studies that varied in the unit of analysis used and noted that after one statistically controls for 
the characteristics of students, the effects of attending different types of four-year institutions are 
both small and inconsistent. However, instead of concluding that different types of four-year 
institutions have essentially the same impact on student development, they offer an alternative 
explanation for the absence of institutional effects. They argue that student precollege 
characteristics are not independent of the institution attended, and that global college 
environment measures may have little impact on students given the sub environments existing 
within institutions such as different majors and living arrangements that are more proximal to 
students’ daily experiences. Essentially, Pascarella and Terenzini are acknowledging the 
multilevel nature of postsecondary institutions and its impact on research on college effects. 

As Pascarella and Terenzini (1991) have documented, the unit of analysis issue has been 
a complex and controversial issue in the research on the influence of college. The problem with 
disaggregating higher order variables to the individual level is the fact that if students are in the 
same institution then they have the same value on each institutional variable. Individuals in one 
group, whether it be individuals in the same major or individuals attending the same institution, 
are more similar than individuals in different groups. Thus students in different majors or 
institutions can be independent but students in the same majors or institutions share values on 
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many more variables. Some of these variables are not observed, which means they vanish into 
the error tenn of the linear model, causing correlations between disturbances. The sharing of the 
same group is a likely cause of dependency among observations. To acknowledge the 
dependency of these individuals is important because it changes the error variance in traditional 
OLS regression. The error variance in traditional OLS regression represents the effect of all 
omitted variables and measurement error, under the assumption that these errors are unrelated. 
The degree of covariance in the error terms of individuals sharing the same institution or 
academic major is expressed in the intra-class correlation coefficient. OLS regression fails to 
capture the positive intra-class correlations that results from the interdependencies among 
students within the same institution, major, class, etc. These interdependencies are brought about 
by the common experiences of students within the same institution or because of the ways in 
which students were initially drawn to an institution and result in misestimated standard errors 
(Burstein, 1980). 

In the second approach, student characteristics are aggregated over institutions and an 
institutional analysis is done. The procedure forces the researcher to assume that all individuals 
within the same institution are affected identically by the institutional level characteristics. 
Conceptually, this is an obvious error since institutions allocate financial resources differently 
through the institution. The main problem with this approach is the loss of the within-group 
information, which can usually account for up to 80 to 90% of the total variation (Burstein, 1980; 
Ethington, 1997; Hannan & Burstein, 1974; Kreft & de Leeuw, 1998). As a consequence, 
relations between aggregated variables are often much stronger and can be quite different from 
the relation between those at the individual level (Burstein, 1980; Draper, 1995; Kreft & de 
Leeuw, 1998). The aggregation approach is also problematic conceptually because student level 
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characteristics change their meaning when aggregated; this is known as aggregation bias. 
Aggregation bias occurs when a variable takes on different meaning and therefore may have 
different effects at different levels of analysis. 

For example, when analyzing workers in 12 different industries, Kreft, de Leeuw, and 
Aiken (1995) drew contradicting conclusions based on differing units of analysis. In their first 
analysis, executed at the level of the individual worker, the data showed a positive relationship 
between educational level and income: the higher the educational level, the higher the personal 
income. In the second analysis, executed at the level of the industry, the data showed a negative 
relationship between education and income: the higher the average educational level of an 
industry, the lower the average income of workers in that industry (colleges and universities are a 
good example of this). The industry-level analysis used aggregated measures, and these results 
illustrate that analyses executed at different levels of the hierarchy do not necessarily produce the 
same results. The fact that aggregate measures analyzed at the higher level of the hierarchy can 
produce results different from the original individual results has been well documented 
(Burstein, 1980; Robinson, 1950). An important conclusion can be drawn from these results. 
Since educational attainment had a positive effect on income if the unit of analysis is the 
individual and a negative effect on income if the unit of analysis is the industry, the logical 
conclusion is that the variable education measures different things depending on the unit of 
analysis. As Burstein (1980) argues, the issue is not that one unit of analysis is more appropriate 
than the other; rather the issue should be understood in light of the fact that different units of 
analysis are asking different questions of the data. 

Therefore, a statistical analysis that can take into account the problems associated with 
the unit of analysis problem and model all levels of interests simultaneously is needed. 
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Hierarchical linear models (HLM) solve the problems associated with the unit of analysis 
problem such as misestimated standard errors, heterogeneity of regression and aggregation bias 
by modeling all levels of interest simultaneously. Hierarchical linear modeling resolves the 
problem of misestimated standard errors by incorporating a unique random effect for each 
institution into the statistical model; moreover, the variability in these random effects is taken 
into account in estimating the standard errors (Raudenbush & Bryk, 2002). Until the advent of 
HLM, heterogeneity of regression had often been viewed as a methodological nuisance. 

However, the cause of heterogeneity of regression is often of substantive interest. HLMs enable a 
researcher to estimate a separate set of regression coefficients for each higher level 
organizational unit and then model variation among the higher level units in their sets of 
coefficients as multivariate outcomes to be explained by higher level factors (Raudenbush & 
Bryk, 2002). HLMs solve the problem of aggregation bias by modeling each level of the 
hierarchy with its own model. 

Today, many higher education scholars are rushing to use this new, sophisticated analytic 
procedure (Smart, 2005). This rush seems to be based on the assumption that HLM might yield 
substantively different findings than those from studies based on OLS regression analyses. With 
this in mind, the current study investigates the different conclusions that may be drawn 
depending upon the type of analysis chosen. This paper focuses on the three types of analyses 
discussed above. The first analysis will be an OLS regression with the student as the unit of 
analysis, the second analysis will be an OLS regression with the student level variables 
aggregated to the institutional level with the institution as the unit of analysis, and the third 
analysis will be a three-level hierarchical linear model with student characteristics modeled at 
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Level 1, characteristics about the major modeled at Level 2 and characteristics of the institution 
modeled at Level 3. 

METHODOLOGY 

Sample 

Since maximum likelihood estimation methods used in hierarchical linear modeling are 
asymptotic, sample sizes must be sufficiently large. An important issue in multilevel modeling is 
what constitutes a sufficient sample size for accurate estimation and the associated standard 
errors. The main problem is usually the sample size at the group level, because group-level 
sample size is always smaller than the individual-level sample size. Simulation studies have been 
used to address this problem. A review of the few simulation studies that have been carried out to 
date suggest that a large number of groups is generally more important than a large number of 
people per group (Maas & Hox 2004, 2005). The absolute minimum number of groups for 
accurate maximum likelihood estimation is debatable. Kreft and de Leeuw (1998) recommend 
that 30 groups is the absolute smallest acceptable number of groups for an HLM analysis. Maas 
and Hox (2004, 2005) recommend no less than 50 groups. They have shown that when the 
number of groups is less than 50 and group size is less than 30 the standard errors of the Level-2 
variance components are biased downward. In addition, Maas and Hox (2004) recommend the 
following rule of thumb: “if one is only interested in the fixed effects of the model, ten groups 
can lead to good estimates. If one is also interested in contextual effects, 30 groups are needed. If 
one also wants correct estimates of the standard errors, at least 50 groups are needed” (p. 135). 

Data for this study were taken from the 2006 administration of the National Survey of 
Student Engagement (NSSE). The NSSE obtains information from a random sample of first-year 
and senior students about the nature of their undergraduate experiences and measures the extent 
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to which students engage in effective educational practices (Kuh, 2001). The institutions that 
participated in NSSE 2006 were similar to the national profile of all baccalaureate degree- 
granting institutions in the United States in terms of sector, geographic region, and urban-rural 
locale. 

The sample used in this study consists of senior students who completed the NSSE 
survey in 2006. Only students who had begun college at their current institution were selected for 
the sample. The restriction to include only students who had begun at their current institution 
was made in order to examine institutional effects. Students that had transferred to their current 
institution may not have had time to gauge important contributions of the institution. Next, 
institutions and majors with less than 30 students were omitted from the sample. This restriction 
was made in reference to Maas and Hox’s (2004, 2005) recommendations on appropriate sample 
sizes for HLM analyses. The final sample used in this study consists of 56,276 senior students in 
58 majors from 405 U.S. institutions that started college at their current institution and who had 
complete data on the variables described below. 

Variables 

For the purpose of this study, the dependent variable is self-perceived growth in critical 
thinking skills and is perceived to be a function of student attributes, the influence of the 
student’s major, and attributes of the institution they attend. The variables chosen for this study 
to operationalize student attributes, college major attributes, and institutional attributes were 
selected from Pascarella and Terenzini’s (2005) review of the literature of college effects on 
students. The student characteristics hypothesized to impact critical thinking are three scales 
measuring course emphasis on higher-order thinking skills, students’ level of academic effort, 
and student-faculty interaction. Two of Biglan’s (1973a, 1973b) three dimensions, hard vs. soft 
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and pure vs. applied, were used to measure influences from the major. The characteristics of the 
institution hypothesized to impact critical thinking are measures of students’ perceptions of 
supportive campus environment, the selectivity of institution, the extent of graduate emphasis, 
and the residential character of the institution. 

The variables used in this study were constructed from items included in the 2006 
administration of the NSSE survey, Biglan’s (1973a, 1973b) classification of academic 
disciplines, the 2005 Carnegie advanced classification, and Barron’s ratings of institutional 
selectivity. The dependent variable used in the analyses was a scale representing student’s 
perceived critical thinking ability (CT). The NSSE survey asked students questions regarding the 
extent to which their experiences at their current institution contributed to their knowledge, 
skills, and personal development in thinking critically and analytically, analyzing quantitative 
problems, and solving complex real-world problems. The alpha reliability coefficient for 
perceived critical thinking ability was 0.79. Appendix A provides a complete list of the items 
comprising each variable along with the coding and construction procedures. 

The student characteristics hypothesized to impact critical thinking are three scales 
representing course emphasis on higher-order thinking skills, academic effort, and student- 
faculty interaction. The six items comprising the course emphasis on higher-order thinking skills 
scale (HOT) ask students the extent to which their coursework emphasized analyzing and 
synthesizing ideas, making judgments, and applying theories. The 1 1 items comprising the level 
of academic effort scale (AE) ask students questions related to course rigor and preparation. The 
five items comprising the student-faculty interaction scale (SFI) ask student about discussions 
and interactions with faculty members. Alpha reliability coefficients for these scales are 0.80, 
0.67, and 0.77, respectively. The selection of items for these scales was taken from Pike, Kuh, 
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and McCormick (2008); in their study, Pike et al. found alpha reliability coefficients similar to 
the ones found in this study. 

The major characteristics hypothesized to impact critical thinking are Biglan’s (1973a, 
1973b) hard versus soft dimension and pure versus applied dimension. The hard versus soft 
dimension (HARD) reflects the degree to which an academic discipline possesses a clearly 
delineated paradigm. The pure versus applied dimension (PURE) reflects the academic 
discipline’s concern with practical application. Each major was classified as either hard or soft 
and either pure or applied. Thus, for example mathematics is classified as both “Hard” and 
“Pure” whereas finance is classified as both “Soft” and “Applied.” The hard versus soft 
dimension is coded 0 for soft disciplines and 1 for hard disciplines. The pure versus applied 
dimension is coded 0 for applied disciplines and 1 for pure disciplines. Appendix B lists all the 
majors and their Biglan classification. 

The institutional characteristics hypothesized to impact students’ perceived critical 
thinking are measures of the supportive campus environment, the selectivity of institution, the 
extent of graduate emphasis, and the residential character of the institution. The six items 
comprising the supportive campus environment scale (SCE) ask students questions about their 
institutions commitment to their academic and social success and their relationships with other 
students, faculty members, and administrative personnel. Since the supportive campus 
environment scale is considered an institutional characteristic it was aggregated for each 
institution. The supportive campus environment scale represents a characteristic of the normative 
institutional environment and is the average perception of the supportive environment of the 
institution. Alpha reliability coefficient for the supportive campus environment scale is 0.78. The 
selection of items for the supportive campus environment scale was also based on Pike et al. 
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(2008). Furthermore, the alpha reliability coefficient computed by Pike et al. was similar to the 
one found in this study. 

In addition, two of the 2005 Carnegie advanced classifications were used. The first 
classification, graduate coexistence, measures the extent to which an institution awards graduate 
degrees in the same fields in which they award undergraduate degrees. The values in the 
graduate coexistence variable were merged into three categories: no graduate coexistence, some 
graduate coexistence, and high graduate coexistence. Then, two dummy variables were created. 
One dummy variable (SG) was coded 1 for some graduate coexistence and 0 otherwise. The 
other dummy variable (FIG) was coded 1 for high graduate coexistence and 0 otherwise. Thus, 
no graduate coexistence was the comparison variable. The next classification measures the 
institutions’ residential character. The values in the residential character variable were merged 
into three categories: primarily commuter, primarily residential, and highly residential campuses. 
Then, two dummy variables were created. One dummy variable (PC) was coded 1 for primarily 
commuter and 0 otherwise. The other dummy variable (PR) was coded 1 for primarily residential 
and 0 otherwise. Thus, highly residential was the comparison variable. The final institutional 
characteristic used in this study is the 2005 Barron’s ratings of institutional selectivity (BAR). 
This index has 1 1 categories ranging from “noncompetitive” to “most competitive.” 

Data Analysis 

This study estimated the effects of the above variables in three different ways. The first 
analysis was an OLS regression with the student as the unit of analysis. The second analysis was 
an OLS regression with the institution as the unit of analysis. The third analysis was a three-level 
HLM analysis with student characteristics modeled at Level 1 , characteristics of the academic 
discipline modeled at Level 2, and characteristics of the institutions modeled at Level 3. Prior to 
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the estimation of the two OLS regression models and the HLM model, exploratory analyses were 
conducted testing the assumptions underlying each of the analyses. Nonnal probability and 
residual plots indicated that the OLS regression assumptions of nonnality and heterogeneity of 
variance were satisfied. Moreover, residual statistics were checked for any potential outliers and 
influential data points. Variance inflation factors (VIF) were calculated and results indicated that 
multicollinearity was not an issue in the data. 

For the three-level HLM analysis, model assumptions were checked by comparing the 
results of the model-based fixed effects with the results of the fixed effects with robust standard 
errors. Since the number of Level-3 units is relatively large, the model-based fixed effects can be 
compared to the fixed effects with robust standard errors. If the model-based fixed effects and 
the fixed effects with robust standard errors differ substantially, it suggests problems with 
nonnality, homosecdasticity, or linearity (Raudenbush & Bryk, 2002). In this analysis, model- 
based fixed effects and fixed effects with robust standard errors were similar suggesting no 
severe violations of the assumptions underlying the application of hierarchical linear modeling. 

The first analysis was an OLS regression with the student as the unit of analysis. The 
equation estimated was 

CT = p 0 + Pi (HOT) + p 2 (AE) + p 3 (SFI) + p 4 (HARD) + p 5 (PURE) + p 6 (SCE) + p 8 (SG) + p 9 

(HG) + pio (PC) + pn (PR) + P12 (BAR) + s 

In this analysis all variables were measured at the student-level except the supportive campus 
environment scale, which was aggregated to the institution-level. The second analysis was an 
OLS regression with the institution as the unit of analysis. The equation estimated was the same 
as the previous equation except that all variables were aggregated to the institution-level. 
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Appendix C and D present the means, standard deviations, and correlations used for estimating 
the two OLS regression models. 

The third analysis was the three-level HLM model. The HLM analysis was conducted in 
four phases. The first phase begins by estimating a model that has no Level- 1, Level-2, or Level- 
3 predictors. The purpose of estimating a model with no predictors was to represent how the 
variation in students’ perceived critical thinking ability was allocated across the three different 
levels (student, major, and institution). Raudenbush and Bryk (2002) refer to this model as the 
fully unconditional model since there are neither student-level predictors used at Level 1 or any 
major or institutional characteristics as predictors at Level 2 or Level 3. The fully unconditional 
three-level model partitions the total variability in critical thinking skills into its three 
components: among students within majors, a ; among majors within institutions, z n ; and among 
institutions, ip. It also allows for the estimation of the proportion of variation that is within 
majors, among majors within institutions, and among institutions. 

In the second phase, a full Level- 1 model was estimated using the students’ 
characteristics to predict students’ perceived critical thinking ability. Within each major, 
students’ perceived critical thinking ability was modeled as a function of student-level predictors 
plus a random student-level error. In order to better compare the results from the HLM analysis 
and the two OLS regression models, slope effects in the HLM analysis were constrained to be 
fixed. Thus, in the HLM analysis only the intercepts were allowed to vary across majors and 
institutions. Considering students’ perceived critical thinking ability to be a function of course 
emphasis on higher-order thinking skills, academic effort, and student-faculty interaction, the 
following equation was estimated for each major: 

CT jjk T^Ojk T n ljk (ROT ijk ) T n 2jk (AE yk) H3jk (SFIy^) T C/y^ 
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Each of the student-level predictors were centered about the major mean, and thus, 
nojk is the average across majors; 

n ijk is the effect of course emphasis on higher-order thinking skills on critical thinking; 

K 2 jk is the effect of academic effort on critical thinking; 

n 3 jk is the effect of student-faculty interaction on critical thinking; and 

Qijk is the student-level random effect that represents the deviation of student ijk ’s score 

from the predicted score based on the student-level model. 

The third phase tested whether the effects of the intercept, nojk, varied across majors. 
Since significant variability was found, nojk was modeled as a function of the two Level-2 
variables. In order to better compare the results of the HLM analysis with the two OLS 
regression analyses, the slope effects were fixed to equal the average across majors; only the 
intercept was allowed to vary. Thus, the following equation was estimated for each major: 

Kojk = \iook + (HARD) + P 02 k (PURE) + xojk 

where 

finok is the intercept for institution k in modeling the major effect 7 to,*; 

P oik is the corresponding coefficient that represents the direction and strength of 
association between major characteristic (HARD) and nojk, 

P 02 k is the corresponding coefficient that represents the direction and strength of 
association between major characteristic (PURE) and 7%*; and 
r ojk is a Level-2 random effect that represents the deviation of major jk’s Level- 1 
coefficient, nojk, from its predicted value based on the major-level model. 

The final phase tested whether the effects of the intercept, P ook, varied across institutions. 
Since significant variability was found, $ook was modeled as a function of the four Level-3 
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variables. The slopes were fixed to equal the average across institutions; moreover, the 
supportive campus environment scale and Barron’s rating of institutional selectivity variable 
were entered into the model centered around the grand mean. Thus, the following equation was 
estimated for each institution: 

$ook = y ooo + y ooi (SCE) + yoo 2 (SG) + y oos (HG) + y 004 (PC) + y oos (PR) +y 006 (BAR) + u ook 
where 

y ooo is the intercept term in the institution-level model for fiook', 

y oos, where s = is the corresponding Level-3 coefficient that represents the direction and 
strength of association between the institution characteristic and $ook, and 
U-ook is a Level-3 random effect that represents the deviation of school k’s coefficient, P ook, 
from its predicated value based on the institution-level model. 

RESULTS 

Student-level OLS Regression 

In the first analysis, ordinary least squares regression was used to examine the influences 
on students’ perceived critical thinking ability. The regression results indicated that the set of 
independent variables explained 3 1 .9% of variance in critical thinking (F ( 1 1 , 56264) = 2396.09, 
p < .001). Regression results are given in Table 1. In the presence of the other variables in the 
model, the variables which had a significant, unique relationship with students’ perceived critical 
thinking ability were course emphasis on higher order thinking skills ( b = 0.394), academic effort 
( b = 0.144), student-faculty interaction ( b = 0.1 10), hard vs. soft dimension ( b = 3.028), pure vs. 
applied dimension ( b = -2.216), perceptions of supportive campus environment ( b = 0.075), high 
graduate coexistence ( b = 0.759), primarily commuter institutions ( b = 1.291), and primarily 
residential institutions ( b = 0.760). Only two variables did not have a significant impact on 
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perceived gains in critical thinking: some graduate coexistence, and Barron’s ratings of 
institutional selectivity. 

Institution-level Regression 

The second analysis was an OLS regression with the institution as the unit of analysis. In 
this analysis, all variables were aggregated to the institution-level. The regression results 
indicated that the set of independent variables explained 66.2% of variance in critical thinking ( F 
(11, 393) = 69.99, p < .001). Regression results are given in Table 2. In the presence of the other 
variables in the model, the variables which had a significant, unique relationship with perceived 
critical thinking ability were course emphasis on higher order thinking skills ( b = 0.485), 
academic effort ( b = 0.192), hard vs. soft dimension ( b = 4.047), pure vs. applied dimension ( b = 
-2.726), perceptions of supportive campus environment ( b = 0.075), primarily commuter 
institutions ( b = 1.328), and primarily residential institutions ( b = 0.674). Four variables did not 
have a significant influence on students’ aggregated perceived critical thinking ability: student- 
faculty interaction, some graduate coexistence, high graduate coexistence, and Barron’s ratings 
of institutional selectivity. 

Hierarchical Linear Modeling Analysis 

The third analysis was a three-level hierarchical linear model with student attributes 
modeled at Level 1, attributes of the major modeled at Level 2, and institutional characteristics 
modeled at Level 3. The three-level hierarchical linear model was analyzed using HLM 5.05 
(Raudenbush, Bryk, & Congdon, 2000). The first step in the hierarchical linear modeling process 
involved determining how the variation in critical thinking was distributed among the three 
different levels: student, major, and institution. This was accomplished by estimating the fully 
unconditional model with no predictors at any of the three levels. Table 3 gives the results of the 



AIR 201 1 Forum, Toronto, Ontario, Canada 




Analyzing multilevel data 1 8 



estimation of the fully unconditional model. The estimation of the grand mean of critical 
thinking across all majors within all institutions (the fixed effect) is 49.97. Decomposing the total 
variability in critical thinking into its’ three components the estimates for the variability among 
students within majors (c~), among majors within institutions (x^), and among institutions (jtp) 
are 91.586, 5.075, and 2.510, respectively. Using these parameter estimates the intra-class 
correlations can be calculated. In this case, the proportion of variance among students within 
majors was 92.4%, the proportion of variance among majors within institutions was 5.1%, and 
the proportion of variances among institutions was 2.5%. Chi-square test indicate that critical 
thinking scores vary significantly among majors within institutions (/ (9036) = 1 1965.90 ,p < 
.001) and vary significantly among institutions (j (404) = 1 298.32, p < .001). This variability 
will subsequently be modeled by using characteristics of the majors to predict %ojk (student-level 
intercept) and institutional measures to predict P ook (major-level intercept). 

In the second step, a full Level- 1 model was estimated using the students’ characteristics 
to predict students’ perceived critical thinking ability. In this analysis, slope effects of the 
student-level variables were fixed to be equal to average across majors within institutions. All 
three student-level variables were centered around their respective group means, so that the 
intercept, nojk, would represent the average critical thinking score across majors within 
institutions. This step was perfonned in order to estimate the proportion of variance in critical 
thinking ability among students within majors explained by the addition of the student-level 
predictors. The addition of the student-level variables (course emphasis on higher order thinking 
skills, academic effort, and student-faculty interaction) explained 27.7% of student-level 
variance. Additionally, the chi-square test revealed significant variation in the intercept, %ojk. 
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across majors (j 2 (9036) = 15721.68,/? < .001). Table 4 gives the results of the HLM estimation 
of random effects for the Level- 1 model. 

Since the intercept, nojk, varies across majors, it was modeled as a function of the Level-2 
variables. In this analysis, student-level slopes were fixed to equal the average across majors 
within institutions and the major-level slopes were fixed to equal the average across institutions. 
Both major-level variables were entered into the model uncentered, so that the intercept, Poofo 
represents the average critical thinking score across majors within institutions. The major-level 
variables were entered uncentered because they were dichotomies unlike the continuous 
variables at the student-level. The addition of the major-level variables (hard vs. soft dimension 
and pure vs. applied dimension) explained 21.88% of the variance in the student-level intercept, 
Kojk- In other words, the Biglan (1973a, 1973b) variables, hard versus soft dimension and the pure 
versus applied dimension, explained 21.88% of variance between majors. Chi-square test 
indicate that the remaining unexplained variability is still significant (/ (9034) = 14630.10 ,/? < 
.001) indicating that there still are significant differences among the mean critical thinking levels 
of majors not explained by the two Biglan variables. Additionally, the chi-square test revealed 
significant variation in the major-level intercept, $ook(x~ (404) = 1379.57,/? < .001) across 
institutions. Table 5 gives the results of the HLM estimation of random effects for the Level-2 
model. 

In the final step, the full HLM analysis was modeled. Since the intercept, $ook, varies 
across institutions, it was modeled as a function of the Level-3 variables. In this analysis, 
student-level slopes were fixed to equal the average across majors within institutions and major- 
level slopes were fixed to equal the average across institutions. In this analysis, the supportive 
campus environment scale and Barron’s ratings of institutional selectivity were entered into the 
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model centered around their respective grand means and the other variables were entered into the 
model uncentered, so that the intercept, yooo, represents the average critical thinking score across 
institutions. Results for the random effects are found in Table 6, and results for the fixed effects 
are found in Table 7. The addition of the institution-level variables explained 57.66% of the 
variance in major-level intercept, fiook- In other words, the addition of the institutional 
characteristics explained 57.66% of the variance between institutions. Chi-square test indicate 
that the remaining unexplained variability is still significant (x (398) = 858. 1 7, p < .001) 
indicating that there still are significant differences among the mean critical thinking levels of 
institutions not explained by the six institutional characteristics. 

The final estimation of the fixed effects was as follows. All three student-level variables 
had a significant impact on students’ perceived critical thinking ability. The estimated effect for 
course emphasis on higher order thinking skills was 0.387; the estimated effect for academic 
effort was 0.136, and the estimated effect of student-faculty interaction was 0. 125. Both major- 
level variables had a significant impact on average critical thinking across major within 
institution. The estimated effect for the hard vs. soft dimension was 2.655, and the estimated 
effect for the pure vs. applied dimension was -2.111. Four of the six institution-level variables 
had a significant effect on the average critical thinking across institutions. The estimated effect 
for supportive campus environment was 0. 140; the estimated effect for primarily residential was 
0.584; the estimated effect for primarily commuter was 1 .3 10, and the estimated effect for 
Barron’s ratings of institutional selectivity was 0.401. The estimated effect for the intercept, yooo, 
was 50.21 1. 

DISCUSSION 
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The primary interest of this study was to investigate the differences in substantive 
conclusions one may come to depending upon the type of analysis chosen, OLS regression or 
HLM. Thus, empirical data were employed in this study. By using empirical data, we are dealing 
with a more realistic research situation instead of a robustness study where data are computed 
based on fixed parameters then altered to meet certain criteria. Using empirical data, instead of 
data computed based on certain parameters, places this study in the literature of college impact 
studies, and in doing so, we are better able to test the theoretical framework from the higher 
education literature. When we use OLS regression, the statistical model does not fit the nature of 
the data used when investigating the influence of college on students. On the other hand with 
HLM, the statistical model fits the theoretical model where students are nested within majors 
nested within institutions. Thus, by using empirical data, we were able to examine whether a 
misspecified statistical model, such as OLS regression, can produce parameter estimates 
comparable to a statistical model that better represents the theoretical model under study, such as 
HLM. Since this is empirical data, we are unable to know the true parameter estimate. However, 
if we acknowledge that HLM provides the best statistical model, and as a result, gives us the best 
parameter estimates, we can investigate how parameter estimates produced by HLM compare 
with the estimates produced by OLS regression. 

OLS Regression with the Student as the Unit of Analysis 

In the field of higher education, researchers studying college effects on students generally 
use the student as the unit of analysis. Often times, these studies contain mixed forms of data. 
Researchers acknowledge the importance of the hierarchical nature of the organization of 
postsecondary education, which is why they typically collect infonnation about students and 
characteristics of the institutions they attend. Thus, if we perfonn the analysis as the majority of 
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higher education researchers would, we would come to the following conclusions regarding the 
influences on students’ perceived critical thinking ability. 

Results from the OLS regression with the student as the unit of analysis (Table 1) 
indicate that the set of independent variables explain 3 1 .9% of the variance in students’ 
perceived critical thinking ability. From the results, we see that all three student attributes have a 
significant impact on students’ perceived critical thinking ability; furthermore, course emphasis 
on higher order thinking skills has the strongest relationship to students’ perceived critical 
thinking ability ( b = 0.394, [> = 0.394). Thus, coursework that emphasizes analyzing, 
synthesizing, and making judgments about ideas and information, applying theories or concepts 
to new situation, integrating ideas from various sources of information, and putting together 
ideas or concepts from different courses when completing assignments, leads to higher 
perceptions of critical thinking ability than coursework that does not emphasize these types of 
learning. The second strongest relationship was students’ level of academic effort ( b = 0.144, ft = 
0.144). The greater investment of time and effort students put into their academic work, the 
greater the perceived gains in critical thinking. Student-faculty interaction (Z? = 0.110,/? = 0.111) 
is also shown to have a positive, significant relationship to students’ perceived critical thinking 
ability indicating that the more time and effort students spend interacting with faculty members 
the greater the perceived gains in critical thinking. 

Both major characteristics have a significant influence on students’ perceived critical 
thinking ability. Students majoring in hard fields ( b = 3.028, /? = 0. 13 1), i.e., academic 
disciplines that have a commonly agreed upon set of problems for study and accepted methods 
for exploring these problems, tend to perceive greater critical thinking ability than students 
majoring in soft fields. Furthermore, students majoring in applied fields ( b = -2.216, (> = -0. 1 1 1), 
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i.e., an academic discipline that is concerned with the practical application of its subject material, 
tend to perceive greater critical thinking ability than students majoring in pure fields. 

The following institutional characteristics have a significant influence on students’ 
perceived critical thinking ability: supportive campus environment ( b = 0.075, p = 0.065), high 
graduate coexistence ( b = 0.759, P = 0.034), primarily commuter institutions (b = 1.291, P = 
0.056), and primarily residential institutions ( b = 0.760, p = 0.036). The greater students 
perceived the campus as a supportive and friendly place the greater the perceived gains in critical 
thinking. Additionally, students attending institutions with a high graduate coexistence perceive 
greater gains in critical thinking than student who attend institutions with no graduate 
coexistence. Interestingly, students that attended institutions that were not highly residential 
institutions, in other words, institutions that were primarily commuter or primarily residential, 
perceive greater critical thinking ability than students that attended intuitions that were highly 
residential. Selectivity, as measured by Barron’s ratings of institutional selectivity, was not found 
to have a significant relationship with students’ perceived critical thinking ability. 

Although the supportive campus environment scale, high graduate coexistence and 
residential character variables have a significant influence on students’ perceived critical 
thinking ability, the statistical significance could be due to the large sample size (n = 57,276) 
used in the analysis. While this large sample size was not required for the OLS regression model, 
it was needed in order to meet the appropriate sample size requirements recommended for HLM 
analyses by Maas and Hox (2004, 2005). In light of the large sample size, Pedhazur (1997) 
argues that standardized regression coefficients smaller than 0.05, regardless of probability level, 
are substantively not worth interpreting. Pedhazur reasons that when sample size is relatively 
large, even substantively meaningless regression coefficients may be statistically significant. 
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Consequently, researchers should use a criterion of meaningfulness, specific to the area of study, 
when interpreting significant regression coefficients. Given Pedhazur’s argument and the reality 
of the large sample size, it is reasonable to assume that, according to this analysis, the effects of 
institutional characteristics are minimal at best. 

A Comparison of the Two OLS Regression Models 

An important question when investigating the influence of college on students is the 
appropriate unit of analysis. Generally, higher education researchers use the student as the unit of 
analysis when studying college effects on students. An alternative approach, prior to more 
advanced statistical techniques, was to aggregate the student-level data to the institution-level 
and perfonn the analysis on the institution. If this approach were taken to analyze the data, we 
would have come to the following conclusions regarding the influences on the average student’s 
perceived critical thinking ability. 

Results from the OLS regression with the institution as the unit of analysis (Table 2) 
indicate that the set of independent variables explain 66.2% of the variance in the average 
student’s perceived critical thinking ability. In the institution OLS regression model, two of the 
three student characteristics have a significant influence on the average students’ perceived 
critical thinking ability. Just as in the previous analysis, course emphasis on higher order 
thinking skills ( b = 0.485, ft = 0.499) has the greatest impact on critical thinking ability. In 
addition, students’ level of academic effort (b = 0. 192, ft = 0.207) also has a significant impact 
on critical thinking. Different from the student level analysis, student-faculty interaction was not 
shown to significantly impact the average students’ critical thinking ability. 

Again, both major characteristics have a significant influence on the average students’ 
perceived critical thinking ability. Students majoring in hard fields ( b = 4.047, ft = 0.322), tend to 
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perceive greater critical thinking ability than students majoring in soft fields. Furthermore, 
students majoring in applied fields ( b = -2.726, f> = -0.248) tend to perceive greater critical 
thinking ability than students majoring in pure fields. The institutional characteristics that have a 
significant influence on the average student’s perceived critical thinking ability are the same as 
in the OLS regression analysis with the student as the unit of analysis except for the effect of 
high graduate coexistence. Institutions where students perceive the campus as a supportive and 
friendly place tend to have average student bodies that also perceive greater critical thinking 
ability {b = 0.075, /? = 0.322). Moreover, the average student body at institutions that are 
primarily commuter ( b = 1.328 = 0.271) or primarily residential ( b = 0.674, /l = 0.149) perceive 
greater gains in critical thinking than the average student body at institutions that are highly 
residential. In the institution OLS regression model, the two graduate coexistence variables and 
selectivity do not have a significant relationship with the average student’s perceived critical 
thinking ability. 

It was noted that in the student OLS regression analysis, the institutional variables had 
marginal effects. Conversely, in the institution OLS regression analysis all significant 
standardized coefficients were very strong. Moreover, the standardized coefficients for all 
significant variables were larger in the institution OLS regression model than in the student OLS 
regression model. The stronger relationships found in the institution OLS regression analysis 
were expected given the citations in the literature (Burstein, 1980; Draper, 1995; Kreft & de 
Leeuw, 1998) that relations between aggregated variables are often times stronger. Another 
instance where aggregate data tends to be stronger is in the estimation of the variance explained. 
The estimate of variance explained in the institution analysis appears much larger than the 
amount of variance explained in the student model. 
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Given the stronger relationships in the institution OLS regression model, we would 
expect to see a larger proportion of variance explained. The variance explained in the institution 
analysis appears larger because we ignore the individual variability and only have the variability 
that is between institutions, which is a much smaller proportion. This will become more apparent 
when we look at the variance decomposition in the HLM analysis. Finally, the results of the 
institution OLS regression analysis are not as affected by sample size (n = 407) as they are in the 
student OLS regression model. Thus, the statistical significance of the variables in the institution 
OLS regression is much more reliable. 

HLM Estimates of Variance Components 

The three-level HLM analysis allows us to partition the total variability in students’ 
perceived critical thinking ability into its three components: among students, among majors 
within institutions, and among institutions. Calculations of the intra-class correlation coefficients 
(Table 3) show that 92.4% of the total variance in students’ perceived critical thinking is among 
students, 5.1% is due to differences among majors within institutions, and 2.5% is due to 
differences among institutions. As can be seen, most of the variability is due to individual 
differences. Researchers (Burstein, 1980; Ethington, 1997; Kreft & de Leeuw, 1998; Pascarella 
& Terrenzini, 1991, 2005) studying multilevel structures have observed similar results that most 
of the variability in hierarchical structures are due to within group differences. An important 
feature to note concerning the partition of variability is the variation due to differences between 
majors and differences between institutions. In effect, this shows that majors are more important 
in explaining variance than institutions and provides evidence of the importance of academic 
disciplines. Since the HLM analysis allows us to partition the total variability in students’ 
perceived critical thinking ability into its three parts, which is something OLS regression is not 
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able to do, we see a better picture of how the variation in students’ perceived critical thinking 
ability is distributed with the HLM model than with either regression analysis. 

For the three-level random-intercept only model used in this study, the variance 
components to be considered are the proportion reduction in Level- 1 residual variance (a ), the 
proportion reduction in random variation over majors at Level 2 (x u ), and the proportion 
reduction in random variation over institutions at Level 3 (xp). In this study, the proportion of 
variance explained in the Level- 1 residual variance by the addition of the Level- 1 predictors 
(higher-order thinking skills, academic effort, and student-faculty interaction) is 27.73%. Thus, 
the student characteristics are explaining 27.73% of the 92.4% of total variation among students. 
The proportion of variance explained in the average critical thinking across majors within 
institutions by the addition of the Level-2 predictors (hard vs. soft dimension and pure vs. 
applied dimension) is 21.88%. Thus, the major characteristics are explaining 21.88% of the 5.1% 
of variability that is due to differences between majors. An important feature to note is that the 
variance explained in the average critical thinking across majors within institutions is conditional 
on the specific Level- 1 model, and the variance reduction statistic is only interpretable for 
models with the same Level- 1 model (Raudenbush & Bryk, 2002). 

The proportion of variance explained in the average critical thinking across institutions 
by the addition of the Level-3 predictors (supportive campus environment, some graduate 
coexistence, high graduate coexistence, primarily commuter, primarily residential, and Barron’s 
ratings of institutional selectivity) is 57.67%. Thus, the institutional characteristics are explaining 
57.67% of the 2.5% of the variability that is due to differences between institutions. Again, the 
variance explained in the average critical thinking across institutions is conditional on the 
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specific Level- 1 and Level-2 model, and the variance reduction statistic is only interpretable for 
models with the same Level- 1 and Level-2 model. 

Estimates of variance explained are not directly comparable between HLM and OLS 
regression because in the HLM analysis we have taken the total variability in students’ perceived 
critical thinking ability and separated it into its three parts: among students, among majors within 
institutions, and among institutions. In the HLM analysis, major characteristics can only account 
for variation among major means. That is, only the parameter variation, t k , is explainable. 
Likewise, institutional characteristics can only account for variation among institutional means. 
Again, only the parameter variation, xp, is explainable. In comparison, ordinary least squares 
regression employs the total outcome variability to compute the variance explained statistic, R- 
squared. The variation among students however, reflects individual effects and errors of 
measurement in the outcome both of which are unexplainable by major characteristics and 
institutional characteristics in HLM (Raudenbush & Bryk, 2002). Since these statistics are 
computed in different ways, there is no straightforward comparison of variance explained 
statistics between OLS regression and HLM analysis. Although variance explained statistics are 
not directly comparable between analyses, with the HLM analysis we are better able to see how 
variance is distributed and how variables measured at different levels affects critical thinking. 
Comparison of the HLM Fixed Effects to the Two Regression Models 

Table 8 gives a comparison of the results across all three analyses. From the results of the 
HLM estimates of the fixed effects, we see that all three student-level measures have a 
significant impact on students’ perceived critical thinking ability. As in the two regression 
models, students whose coursework emphasizes higher-order thinking skills perceive greater 
critical thinking abilities ( 7 ty* = 0.387). Again, like the two regression models, academic effort 
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(it 2 jk = 0.136) is shown to have a significant, positive influence on students’ perceived critical 
thinking ability. As in the student OLS regression model, student-faculty interaction (jisjk = 
0.136) is shown to have a unique influence on students’ perceived critical thinking ability. This 
result is different than what is found in the institution OLS regression model which did not show 
student-faculty interaction to have significant effect on critical thinking ability. Consistent with 
the two regression models, results from the HLM analysis show that both major characteristics 
have a significant effect on the average perceived critical thinking ability across majors. Once 
more, students majoring in hard disciplines ($oik = 2.655) and students majoring in applied 
disciplines (fio 2 k =-2.11 1) perceive greater critical thinking abilities. 

In the HLM analysis, the institutional characteristics that have a significant influence on 
the average critical thinking across institutions are perceptions of supportive campus 
environment (y ooi = 0.140), primarily commuter institutions (y 004 = 1.310), primarily residential 
institutions (y 005 = 0.584) and selectivity (y 006 = 0.401). Findings from the HLM analysis that are 
analogous to the findings from the two regression models are the conclusions drawn regarding 
the effects of students’ perceptions of supportive campus environment, the residential character 
of an institution, and some graduate coexistence. As in the two regression models, institutions 
where students perceive the campus as a supportive and friendly place also tend to report higher 
average critical thinking scores. In addition, institutions that are primarily commuter or primarily 
residential tend to have higher average critical thinking scores than institutions that are highly 
residential. Finally, in all three analyses, the effect of some graduate coexistence is not shown to 
have a significant relationship with critical thinking ability. 

A couple of the effects of the institutional characteristics differ across analyses. Most 
notably, the results of the HLM analysis demonstrate that institutions that are more selective, as 
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measured by the Barron’s rating of institutional selectivity, tend to have higher average critical 
thinking scores than institutions that are less selective. This is an interesting finding that we do 
not observe in either regression analysis. Another result that differs across analyses is the effect 
of high graduate coexistence. In the student OLS regression model, institutions with a high 
graduate coexistence are shown to have higher critical thinking scores; however, the effect of 
high graduate coexistence is not significant in either the institution OLS regression model or in 
the HLM analysis. 

Not only did the effects of the independent variables differ across analyses but coefficient 
estimates differ as well. When comparing coefficient estimates of HLM and OLS regression 
procedures, Raudenbush and Bryk (2002) make the case that coefficient estimates in HLM will 
be similar to the estimates in OLS regression, but the estimates of standard errors will tend to be 
biased downward. They contend that generally the coefficient estimates in the student OLS 
regression will be more similar to HLM estimates than estimates in the institution OLS 
regression model, but the degree of agreement between analyses will depend upon the degree of 
imbalance in the group sample sizes. For instance, if the sample sizes are similar for each higher- 
level organization, the coefficient estimates will be the similar. If the sample sizes are not similar 
for all higher-level groups, as it is in this study, coefficient estimates may differ substantially 
across analyses. 

In this study, there was great imbalance in the group sample sizes, which is a common 
trait in multi-institutional studies. Therefore, we would expect coefficient estimates to vary 
across analyses, and for the most part they did. One instance where they do not vary as widely is 
the in the estimates of the student characteristics. Coefficient estimates for the student 
characteristics were fairly similar across all three analyses. In all three analyses the coefficient 
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estimates for higher order thinking skills and academic effort are essentially the same. The only 
student characteristic to differ across analyses is the coefficient estimate for student-faculty 
interaction in the institution OLS regression model. 

For the major characteristics, coefficient estimates were consistent across analyses for the 
pure vs. applied dimension, but varied greatly in the estimate for the hard vs. soft dimension. The 
estimate for the hard vs. soft dimension in the institution OLS regression model ( b = 4.047) is 
more than one and a half times as large as the HLM estimate (fioik - 2.655). On the other hand, 
the HLM estimate and the student OLS regression estimate ( b = 3.028) are more or less similar 
across analyses. Again, the coefficient estimates in the HLM analysis are more similar to the 
results of the student OLS regression model than the institution OLS regression model. 

Across all three analyses, the effects of the institutional characteristics varied widely. The 
two regression analyses produced identical results for the coefficient estimate for the supportive 
campus environment scale ( b = 0.075); however, the estimate from the HLM analysis (y ooi = 
0.140) is almost double. While the coefficient estimates of some graduate coexistence are similar 
across all analyses, essentially no different than zero, the coefficient estimates for high graduate 
coexistence vary greatly from one analysis to the other. Similarly, the estimates of primarily 
commuter institutions are similar across analyses, while the estimates of primarily residential 
institutions vary from one analysis to the other. Finally, the coefficient estimate for selectivity in 
the HLM analysis (y 006 = 0.401) varied greatly from the estimates in the regression models, 
which are virtually zero. 

The differences shown in the major and institutional characteristics can be attributed to 
the unbalanced nature of the data used in this study. One way to avoid these differences, 
according to Raudenbush and Bryk (2002), would be to have a similar number of individuals in 
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each group. However, unless a researcher specifically samples equal numbers of individuals in 
each group, it is rarely the case to find a dataset with an equal number of individuals in each 
higher-level unit, whether it is an equal number of students in various majors or an equal number 
of students in multiple institutions. Thus, researchers will typically find that coefficient estimates 
produced by HLM will differ from the coefficient estimates produced by OLS regression. 

As noted above, Raudenbush and Bryk (2002) indicate that estimates of standard errors 
of the fixed effects will differ across analyses. They maintain that the standard errors produced 
by the student OLS regression model will generally be too small because OLS regression does 
not take into account the fact that lower-level units are not independent and are clustered within 
higher-level units. Nevertheless, this was not the case for the student characteristics. In this 
study, both the student OLS regression model and the HLM model produced basically the same 
estimates for the standard errors of the fixed effects for the student characteristics. Given the 
large sample size (n = 57,276) used in the student OLS regression and the Level- 1 HLM model, 
we would expect the standard errors to be very small, as they were in both analyses. On the other 
hand, the estimates for the standard errors for the major characteristics and the institutional 
characteristics in the student OLS regression model are substantially smaller than the HLM 
estimates, which Raudenbush and Bryk argue will occur. For the Level-2 and Level-3 HLM 
model, the sample size issue is not as critical because the sample size at these levels are 
drastically smaller, n = 9,441 and n = 407, respectively. Thus, the results produced by HLM for 
the standard errors are similar to what Raudenbush and Bryk argue will occur. 

Estimates of standard errors in the institution OLS regression vary considerably when 
compared to the HLM analysis. Standard errors in the institution OLS regression model are 
higher than the HLM estimates for the student and major characteristics. One reason the standard 
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errors in the institution OLS regression analysis are larger for the student and major 
characteristics could be due to aggregation bias since aggregate data have stronger correlations 
and relationships. Another reason for the discrepancy could be due to the differing sample sizes 
used in the HLM analysis across levels. For the student attributes, the Level- 1 HLM analysis 
uses the sample size at the student-level (n = 57,276); thus, since standard errors are a function of 
sample size, we would expect the standard errors for the student characteristics in the HLM 
analysis to be substantially smaller than the ones found in the institution OLS regression analysis 
which used a sample size of n = 407. The same is true for the major characteristics. The HLM 
analysis used a sample size of n = 9,441 while the sample size in the institution OLS regression 
analysis stayed constant (n = 407). When we examine the standard errors for the institutional 
effects, we find what Raudenbush and Bryk (2002) argue will occur: standard errors in the 
institution OLS regression model are consistently smaller than the standard errors produced by 
HLM. In this instance, both the HLM analysis and the institution OLS regression analysis are 
using the same sample size (n = 407) to estimate these standard errors. Finally, the estimates of 
standard errors for the OLS regression analysis with the institution as the unit of analysis are 
consistently larger than the standard error estimates for the OLS regression analysis with the 
student as the unit of analysis. Again, one reason the standard errors in the institution OLS 
regression are larger could be to aggregation bias. 

Conclusions 

How college affects students is an important topic of research in the higher education 
literature. Traditionally, higher education researchers have utilized the traditional linear model, 
ordinary least squares (OLS) regression, to aid in their investigation of the influence of college 
on students. However, this traditional approach ignores the multilevel nature of the majority of 
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data used in such studies, which can cause a multitude of problems such as misestimated 
coefficients and standard errors, spurious significant effects, aggregation bias, and heterogeneity 
of regression. Therefore, a statistical technique that can take into account the multilevel nature of 
the organization of postsecondary education, such as hierarchical linear modeling (HLM), is 
needed. 

The findings of this study can be summed up as follows. First, the results of the 
institution OLS regression model cannot be considered reliable. Findings from the institution 
OLS regression model differed substantially from the results of the other two analyses. In the 
institution OLS regression model, student-faculty interaction and selectivity were not found to 
have a significant relationship with the average students’ perceived critical thinking ability as 
was found in the HLM analysis. These are two important findings that are not illustrated in the 
institution OLS regression analysis. If a researcher would have performed this analysis, he or she 
would have concluded that institutions that foster environments that lead to greater interactions 
among faculty members and students does not have a significant impact the average students’ 
perceived critical thinking ability. This is contrary to the abundant literature (Astin, 1993; Kuh & 
Hu, 2001; Pace, 1979, 1984; Tinto, 1987) that has demonstrated the importance of student- 
faculty interaction on students’ growth and development in college. If a researcher had 
performed this analysis, he or she would have also concluded that the selectivity of an institution 
is not related to the average students’ perceived critical thinking ability. However, results from 
the HLM analysis tend to suggest otherwise. In addition to the different substantive conclusions, 
the coefficient estimates and standard errors in the institution OLS regression analysis differed 
substantially from the coefficient estimates and standard errors in the HLM analysis. With such 
contradictory findings in the institution OLS regression analysis, it is expected that one would 
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not come to accurate conclusions regarding the influences on the average students’ perceived 
critical thinking ability with an OLS regression with the institution as the unit of analysis. 

Second, the findings from the student OLS regression model can only be partially trusted. 
Evidence from this study suggests that one can be fairly confident in the results obtained for the 
student and major characteristics. Even when modeling major and institutional characteristics in 
the regression model, one can still trust the results of the student-level variables and the major- 
level variables. In addition, the coefficient estimates for the student characteristics and major 
characteristics are similar to those found in the HLM analysis. Estimates of standard errors are 
similar for the student characteristics but differ for the major characteristics. Thus, if a researcher 
had perfonned an OLS regression with the student as the unit of analysis, the researcher would 
have come to the same conclusions regarding the effects of the student characteristics as he or 
she would have if an HLM analysis were perfonned but risk inflation of type I error rates for the 
major characteristics. 

Where the student OLS regression analysis and the HLM analysis primarily differ are in 
the effects of the institutional characteristics. In the student OLS regression model, I argue that 
the effects of the institutional characteristics are minimal at best given the large sample size and 
relatively small standardized coefficients. Thus, if a researcher was to perform the OLS 
regression with the student as the unit of analysis, he or she would erroneously conclude that the 
institutional characteristics do not have a significant impact on students’ perceived critical 
thinking ability; thus, concluding that different types of four-year institutions have essentially the 
same impact on students’ perceived critical thinking ability. Furthermore, coefficient estimates 
and standard errors for the institutional characteristics in the student OLS regression model were 
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substantially smaller than those in the HLM analysis as expected according the Raudenbush and 
Bryk (2002). 

Third, when institutional effects are of prime importance, one should perform an HLM 
analysis in order to be confident in the results obtained for the institutional effects. As discussed 
earlier, the results from both OLS regression analyses failed to accurately describe the effects of 
the institutional characteristics. Thus, when a researcher is interested in institutional effects, 
which is often the case when studying college effects on students, researchers need to utilize 
HLM procedures in order to be confident in the results. Ordinary least squares regression has 
been the foundation on which college effects studies have been built. However, evidence from 
this sample suggests that ordinary least squares regression is not capable of accurately detecting 
institutional effects in the presence of multilevel data. Given the discrepancy in results across all 
three analyses and the lack of consistency in the literature involving the influence of college on 
students (Pascarella & Terenzini, 1991, 2005), regular use of hierarchical linear modeling may 
be one way to yield more valid and informative findings in the college effects literature. 

Overall, evidence from this sample suggests that OLS regression is limited in its ability to 
produce accurate parameter estimates. As discussed earlier, the OLS regression with the 
institution as the unit of analysis produced parameter estimates that were substantially different 
than those produced by the HLM analysis. Therefore, researchers should use caution when using 
an OLS regression with the institution as the unit of analysis to study college effects on students. 
The OLS regression with the student as the unit of analysis produced parameter estimates similar 
to those found in the HLM analysis for the student and major characteristics; however, the 
parameter estimates for the institutional characteristics differed considerably. Therefore, when 
using an OLS regression with the student as the unit of analysis to study the influence of college 
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on students, researchers can be fairly confident in the parameter estimates of the lower-level 
variables, such as estimates for student characteristics, but must be cautious when interpreting 
the parameter estimates for the higher-level variables, such as institutional characteristics. 

In regards to the findings of this study, I make the following recommendations 
concerning the appropriate analysis in the presence of multilevel data. First, if a researcher has 
only collected data on students, yet still recognizes the multilevel nature of the data, one would 
come to similar conclusions with HLM and OLS regression. Second, if a researcher has data 
collected on multiple levels, i.e., student characteristics, major characteristics, and institutional 
characteristics, results from OLS regression and HLM will differ in regards to higher-order 
variables. The researcher can be fairly confident in their findings regarding the lower-level 
variables but cannot trust findings regarding higher-level variable. In this study, the student OLS 
regression and the HLM analysis produced similar results for the student attributes and major 
characteristics but produced substantively different results for the institutional effects. With this 
in mind, I would caution researchers in their attempt to use ordinary least squares regression to 
discern relationships between institutional variables. Given that hierarchical linear modeling 
more accurately describes the nature of data under investigation, when data are collected at 
multiple levels, and when sample size is adequately large enough, hierarchical linear modeling 
yields the best parameter estimates and can allow for a richer, more thorough investigation of the 
phenomenon under study. 
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Table 1 



Student-level OLS Regression Results 



Independent Variables 


b 


S.E. 


A 


t 


Higher Order Thinking Skills (HOT) 


0.394 


0.004 


0.394 


91.890* 


Academic Effort (AE) 


0.144 


0.004 


0.144 


35.73* 


Student-Faculty Interaction (SFI) 


0.110 


0.004 


0.111 


26.65* 


HARD 


3.028 


0.082 


0.131 


36.84* 


PURE 


-2.216 


0.072 


- 0.111 


-30.70* 


Supportive Campus Environment (SCE) 


0.075 


0.005 


0.065 


15.54* 


Some graduate coexistence (SG) 


-0.024 


0.102 


-0.001 


-0.23 


High graduate coexistence (HG) 


0.759 


0.129 


0.034 


5.89* 


Primarily commuter (PC) 


1.291 


0.119 


0.056 


10.84* 


Primarily residential (PR) 


0.760 


0.096 


0.036 


7.92* 


Barron’s ratings of institutional selectivity 
(BAR) 


0.069 


0.037 


0.008 


1.87 


R-square = 0.319 











*p < .001. 
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Table 2 



Institution-level OLS Regression Results 



Independent Variables 


b 


S.E. 


P 


t 


Higher Order Thinking Skills (HOT) 


0.485 


0.047 


0.499 


10.34* 


Academic Effort (AE) 


0.192 


0.040 


0.207 


4.85* 


Student-Faculty Interaction (SFI) 


0.026 


0.041 


0.034 


0.64 


HARD 


4.047 


0.412 


0.322 


9.83* 


PURE 


-2.726 


0.465 


-0.248 


-5.86* 


Supportive Campus Environment (SCE) 


0.075 


0.009 


0.322 


7.96* 


Some graduate coexistence (SG) 


-0.057 


0.177 


-0.013 


-0.32 


High graduate coexistence (HG) 


0.303 


0.252 


0.054 


1.20 


Primarily commuter (PC) 


1.328 


0.217 


0.271 


6.13* 


Primarily residential (PR) 


0.674 


0.170 


0.149 


3.97* 


Barron’s ratings of institutional selectivity 
(BAR) 


0.027 


0.073 


0.014 


0.37 


R-square = 0.662 











*p < .001. 
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Table 3 



HLM Estimation of Unconditional Model 



Fixed Effects 


Coefficient 


S.E. 


t-ratio 


yooo: average student critical thinking score 


49. 97 


0.097 


514.129* 


Random Effects 


Variance 


DF 


Chi-square 


2 

a : variance among student within majors 


91.586 






t k : variance among majors within institutions 


5.075 


9036 


11965.90* 


xpt variance among institutions 


2.510 


404 


1298.32* 


Intra-class Correlations 


Coefficient 






Proportion of variance among students 


92.4% 






Proportion of variance among majors 


5.1% 






Proportion of variances among institutions 


2.5% 







*p < .001. 



AIR 201 1 Forum, Toronto, Ontario, Canada 




Analyzing multilevel data 44 



Table 4 



HLM Estimation of Random Effects with Student-level Predictors (Level-1 Model) 



Random Effects 


Variance 


DF 


Chi-square 


2 

a : variance among student within majors 


66.193 






x K : variance among majors within institutions 


9.500 


9036 


15721.68* 


xp: variance among institutions 


2.549 


404 


1266.26* 



*p < .001. 
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Table 5 



HLM Estimation of Random Effects with Student-level and Major-level Predictors 
(Level-2 Model) 



Random Effects 


Variance 


DF 


Chi-square 


a : variance among student within majors 


66.352 






variance among majors within institutions 


7.422 


9034 


14630.10* 


xp: variance among institutions 


2.606 


404 


1379.57* 



*p < .001. 
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Table 6 



HLM Estimation of Random Effects with Predictors Modeled at All Three Levels (Level-3 
Model) 



Random Effects 


Variance 


DF 


Chi-square 


2 

a : variance among student within majors 


66.370 






variance among majors within institutions 


7.415 


9034 


15445.10* 


xp: variance among institutions 


1.103 


398 


858.17* 



*p < .001. 
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Table 7 



FILM Estimation of Fixed Effects with Predictors Modeled at All Three Levels 



Fixed Effects 


Coefficient 


S.E. 


t-ratio 


Intercept 


50.211 


0.187 


268.65** 


Level 1: Effects on student critical thinking 








Higher Order Thinking Skills (HOT) 


0.387 


0.005 


80.89** 


Academic Effort (AE) 


0.136 


0.005 


30.20** 


Student-Faculty Interaction (SFI) 


0.125 


0.005 


26.84** 


Level 2: Effects on average critical thinking across majors within institutions 


HARD 


2.655 


0.121 


21.99** 


PURE 


- 2.111 


0.106 


_19 99 ** 


Level 3: Effects on average critical thinking across institutions 




Supportive Campus Environment (SCE) 


0.140 


0.010 


13.87** 


Some Graduate Coexistence (SG) 


-0.470 


0.208 


-2.27 


High Graduate Coexistence (HG) 


-0.054 


0.284 


-0.19 


Primarily Commuter (PC) 


1.310 


0.249 


5.27** 


Primarily Residential (PR) 


0.584 


0.204 


2 . 86 * 


Barron’s ratings of institutions selectivity 
(BAR) 


0.401 


0.080 


5.03** 



*p< .01. **p < .001. 
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Table 8 



Comparison of Results across Analyses 





Student OLS 


Institution OLS 


HLM 


Independent Variables 


b 


b 


Coefficient 




(S.E.) 


(S.E.) 


(S.E.) 


Higher Order Thinking Skills (HOT) 


0.394** 

(.004) 


0.485** 

(.047) 


0.397** 

(.005) 


Academic Effort (AE) 


0.144** 

(.004) 


0.192** 

(.040) 


0.136** 

(.005) 


Student-Faculty Interaction (SFI) 


0.110** 

(.004) 


0.026 

(.041) 


0.125** 

(.005) 


HARD 


3.028** 


4.047** 


2.655** 


(.082) 


(.412) 


(.121) 


PURE 


-2.216** 


-2.726** 


-2.111** 


(.072) 


(.465) 


(.106) 


Supportive Campus Environment (SCE) 


0.075** 

(.005) 


0.075** 

(.009) 


0.140** 

(.010) 


Some graduate coexistence (SG) 


-0.024 

(.102) 


-0.057 

(.117) 


-0.470 

(.208) 


High graduate coexistence (HG) 


0.759** 

(.129) 


0.303 

(.252) 


-0.054 

(.284) 


Primarily commuter (PC) 


1.291** 

(.119) 


1.328** 

(.217) 


1.31** 

(.249) 


Primarily residential (PR) 


0.760** 

(.096) 


0.674** 

(.170) 


0.584* 

(.204) 


Barron’s ratings of institutional selectivity (BAR) 


0.069 

(.037) 


0.027 

(.073) 


0.401** 

(.080) 



*p < .01. **p < .001. 
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Appendix A 

Items comprising the variables used in the analyses and the construction of scales 



Critical Thinking (CT) (a = 0.79) 

Computed by summing across the following three items then converting to a T score: 

• To what extent has your experience at this institution contributed to your knowledge, 
skills and personal development in thinking critically and analytically? 

• To what extent has your experience at this institution contributed to your knowledge, 
skills and personal development in analyzing quantitative problems? 

• To what extent has your experience at this institution contributed to your knowledge, 
skills and personal development in solving complex real-world problems? 

Each item is coded 1 = very little, 2 = some, 3 = quite a bit, 4 = very much. 

STUDENT MEASURES 

Course Emphasis on Higher-Order Thinking Skills (HOT) (a = 0.80) 

Computed by summing across the following six items then converting to a T score: 

• How much as your coursework emphasized analyzing the basic elements of an idea, 
experience, or theory, such as examining a particular case or situation in depth and 
considering its components? 

• How much as your coursework emphasized synthesizing and organizing ideas, 
information, or experiences into new, more complex interpretations and relationships? 

• How much as your coursework emphasized making judgments about the value of 
information, arguments, or methods, such as examining how others gathered and 
interpreted data and assessing the soundness of their conclusion? 

• How much as your coursework emphasized applying theories or concepts to practical 
problems or in new situations? 

• How often have you worked on a paper or project that required integrating ideas or 
information from various sources? 

• How often have you put together ideas or concepts from different courses when 
completing assignments or during class discussions? 

Each item is coded same as above. 

Academic Effort (AE) (a = 0.67) 

Computed by summing across the standardized scores of the following eleven items then 

converting to a T score: 

• How often have you prepared two or more drafts of a paper or assignment before turning 
it in? 
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• How often have you worked harder than you thought you could to meet an instructor’s 
standards or expectations? 

Each of these two items are coded 1 = never, 2 = sometimes, 3 = often, 4 = very often. 

• During the current school year, about how much reading and writing have you done? 

• Number of assigned textbooks, books, or book-length packs of course readings. 

• Number of written papers or reports of 20 pages or more. 

• Number of written papers or reports of between 5 and 19 pages. 

• Number of written papers or reports of fewer than 5 pages. 

Each of these four items are coded 1 = None, 2 = 1-4, 3 = 5-10, 4 = 1 1-20, 5 = More than 20. 

• Mark the extent to which your examinations during the current school year have 
challenged you to do your best work. 

This item is coded 1 = very little to 7 = very much. 

• In a typical week, how many homework problem sets do you complete? 

• Number of problem sets that take you more than an hour to complete. 

• Number of problem sets that take you less than an hour to complete. 

Each of these two items are coded 1 = None, 2 = 1-2, 3 = 3-4, 4 = 5-6, 5 = More than 6. 

• How many hours a week do you spend preparing for class (studying, reading, writing, 
doing homework or lab work, analyzing data, rehearsing, and other academic activities)? 

This item is coded 1 = 0 hours, 2 = 1-5 hours, 3 = 6-10 hours, 4 = 11-15 hours, 5 = 16-20 
hours, 6 = 21-25 hours, 7 = 26-30 hours, 8 = more than 30 hours. 

• To what extent does your institution emphasize spending significant amounts of time 
studying and on academic work? 

This item is coded 1= very little, 2 = some, 3 = quite a bit, 4 = very much. 

Student-Faculty Interaction (SFI) (a = 0.77) 

Computed by summing across the following five items then converting to a T score: 

• How often have you used e-mail to communicate with an instructor? 

• How often have you discussed grades or assignments with an instructor? 

• How often have you talked about career plans with a faculty member or advisor? 

• How often have you discussed ideas from your readings or classes with faculty members 
outside of class? 

• How often have you worked with a faculty member on activities other than coursework 
(committees, orientation, student life activities, etc.)? 

Each is coded 1 = never, 2 = sometimes, 3 = often, 4 = very often. 

MAJOR CHARACTERISTICS 
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Hard vs. Soft (HARD) 

A dichotomous variable coded 0 = soft, 1 = hard. 

Pure vs. Applied (PURE) 

A dichotomous variable coded 0 = applied, 1 = pure. 

INSTITUTIONAL CHARACTERISTICS 

Supportive Campus Environment (SCE) (a = 0.78) 

Computed by summing across the standardized scores of the following six items then 
converting to a T scores: 

• Quality of your relationships with other students. 

Item ranges from 1 = unfriendly, unsupportive, sense of alienation to 7 = friendly, 
supportive, sense of belonging. 

• Quality of your relationships with faculty members. 

Item ranges from 1 = unavailable, unhelpful, unsympathetic to 7 = available, helpful, 
sympathetic. 

• Quality of your relationships with administrative personnel and offices. 

Item ranges from 1 = unhelpful, inconsiderate, rigid to 7 = helpful, considerable, flexible. 

• To what extent does your institution emphasize providing the support you need to help 
you succeed academically? 

• To what extent does your institution emphasize helping you cope with your non- 
academic responsibilities (work, family, etc.)? 

• To what extent does your institution emphasize providing the support you need to thrive 
socially? 

Each is coded 1= very little, 2 = some, 3 = quite a bit, 4 = very much. Since this is considered 
an institutional characteristic, it was aggregated for each institution. 

Graduate Coexistence 

Values in this variable were merged into three categories: no graduate coexistence, some 
graduate coexistence, and high graduate coexistence. Then, two dummy variables were 
created. One (SG) was coded 1 = some graduate coexistence, 0 = otherwise. The other (HG) 
was coded 1= high graduate coexistence, 0 = otherwise. 

Residential Character 

Values in this variable will be merged into three categories: primarily commuter, primarily 
residential, and highly residential. Then, two dummy variables will be created. One (PC) was 
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coded 1 = primarily commuter, 0 = otherwise. The other (PR) was coded 1 = primarily 
residential, 0 = otherwise. 

Barron ’s Ratings of Institutional Selectivity (BAR) 

Has eleven categories ranging from 1 = noncompetitive to 6 = most competitive. 
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Appendix B 



List of Majors and Biglan (1973a, 1973b) classification 



Major 


Pure vs. Applied 


Hard v.s\ Soft 


Art, fine and applied 


Pure 


Soft 


English (language and literature) 


Pure 


Soft 


History 


Pure 


Soft 


Journalism 


Applied 


Soft 


Language and literature (except English) 


Pure 


Soft 


Music 


Pure 


Soft 


Philosophy 


Pure 


Soft 


Speech 


Applied 


Hard 


Theater or drama 


Pure 


Soft 


Theology or religion 


Applied 


Soft 


Biology (general) 


Pure 


Hard 


Biochemistry or biophysics 


Pure 


Hard 


Environmental science 


Pure 


Hard 


Microbiology or bacteriology 


Pure 


Hard 


Zoology 


Pure 


Hard 


Accounting 


Applied 


Soft 


Business administration (general) 


Applied 


Soft 


Finance 


Applied 


Soft 


Marketing 


Applied 


Soft 


Management 


Applied 


Soft 


Business education 


Applied 


Soft 


Elementary/middle school education 


Applied 


Soft 


Music or art education 


Applied 


Soft 


Physical education or recreation 


Applied 


Soft 


Aero-/astronautical engineering 


Applied 


Hard 


Civil engineering 


Applied 


Hard 


Chemical engineering 


Applied 


Hard 


Electrical or electronic engineering 


Applied 


Hard 


Industrial engineering 


Applied 


Hard 


Materials engineering 


Applied 


Hard 


Mechanical engineering 


Applied 


Hard 


General/other engineering 


Applied 


Hard 


Atmospheric science (including meteorology) 


Pure 


Hard 


Chemistry 


Pure 


Hard 


Earth science (including geology) 


Pure 


Hard 


Mathematics 


Pure 


Hard 


Physics 


Pure 


Hard 


Statistics 


Pure 


Hard 


Architecture 


Applied 


Soft 


Urban planning 


Applied 


Soft 


Medicine 


Applied 


Hard 


Nursing 


Applied 


Soft 
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Major 


Pure vs. Applied 


Hard vs. Soft 


Pharmacy 


Applied 


Hard 


Allied health/other medical 


Applied 


Soft 


Anthropology 


Pure 


Soft 


Economics 


Applied 


Soft 


Ethnic studies 


Pure 


Soft 


Geography 


Pure 


Soft 


Political science 


Pure 


Soft 


Psychology 


Pure 


Soft 


Social work 


Applied 


Soft 


Sociology 


Pure 


Soft 


Agriculture 


Applied 


Hard 


Communications 


Applied 


Soft 


Family Studies 


Applied 


Soft 


Kinesiology 


Pure 


Hard 


Criminal justice 


Applied 


Soft 


Public administration 


Applied 


Soft 
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